Streaming, Memory-Limited PCA

In this paper, we consider a streaming one-pass-over-the-data model for Principal Component Analysis (PCA). The input, in this case, is a stream of p-dimensional vectors, and the output is a collection of k, p-dimensional principal components that span the best approximating subspace. Consequently, the minimum memory requirement for such problems is O(kp). Yet the standard PCA algorithm requires us to form the empirical covariance matrix, typically a dense p × p matrix, hence requiring O(p) memory. Although there exist several incremental algorithms that require O(kp) memory, to the best of our understanding, these methods currently do not have known finite-sample performance bounds. That is, in the high-dimensional setting where the number of samples and dimensionality scale together, there is no known provably correct algorithm. This paper considers this simple but important problem. We give what is to the best of our knowledge, the first streaming algorithm requiring only O(kp) memory, that makes a single pass over the data, and whose performance matches the standard batch algorithm up to logarithmic factors.